fix: reject impossible CALYPSO atom choices - #382

You are right that (set(s) - overlap) == 0 compares a set to an int and has never fired — that part of #356's diagnosis is correct, and I confirmed the guard has been dead since it was written in 07df321 (#217, 2024-04-30). But the replacement predicate is not the right test, and turning it on is a net regression: it rejects configurations that work today while still hanging on the ones #356 is about.

The predicate is not the condition the loop needs

overlap is the intersection of all sub-lists, so overlap ⊆ L holds for every L. That makes not (set(L) - overlap) equivalent to set(L) == overlap — "some sub-list equals the global intersection". The loop below needs one element per sub-list, all distinct, which exists exactly when Hall's marriage condition holds. The two are unrelated.

Brute-forcing every family over a 3-symbol universe (399 configs, satisfiability decided by exhaustive search):

sub-lists   total  ok-accept  ok-reject  WRONGLY REJECTED  WRONGLY ACCEPTED
    1          7        0          0            7                0
    2         49       18          3           28                0
    3        343      165         51           82               45
  total      399      183         54          117               45

117 of 399 (29%) are satisfiable configs the guard now rejects. 45 (11%) are impossible configs it still lets through, where the loop hangs exactly as #356 describes.

Run against the real make_calypso_task_group_from_config:

[["Li"]]                              d3ca156~1: SUCCESS            head: ValueError
[["Li","Na","K"],["Na","K"],["K"]]    d3ca156~1: ['Li','Na','K']   head: ValueError
[["Li"],["Li"],["Na","K"]]            head: killed at 8s, exit 137 (still spinning)

Note the first row: every single-sub-list config is now rejected, because with one sub-list the intersection is that sub-list. name_of_atoms: [["Li","Na","K"]] — pick one species at random for a 1-species search — is the simplest use of this feature and it now fails at submit.

And the second row is the example printed in the error message itself. Details inline.

Suggested direction

Hall's condition is exact and cheap at CALYPSO species counts:

from itertools import combinations
sets = [set(s) for s in name_of_atoms]
n = len(sets)
if any(
    len(set().union(*(sets[i] for i in sub))) < k
    for k in range(1, n + 1)
    for sub in combinations(range(n), k)
):
    raise ValueError(
        f"cannot pick {n} distinct species from {name_of_atoms}: "
        "some group of sub-lists has fewer candidates than sub-lists"
    )

I checked this against exhaustive search over the same sweep: zero mismatches. Better still would be to replace the while True rejection sampling with a direct matching that constructs the assignment — then the guard is unnecessary and the hang is impossible by construction.

Merge-order warning, not a change request

PR #405 ("Enable remaining Ruff rules", open) rewrites this same line to any((set(s) - overlap) == 0 for s in name_of_atoms) — a pure map→genexp change that keeps the == 0 bug. git merge-tree confirms a direct conflict on this line. If #405 lands after this and the conflict is resolved carelessly, the always-false guard comes back. Worth sequencing them deliberately. (#383 also touches this file but only conflicts on the test file's insertion point — trivial.)

Not a problem, so nobody re-raises it

The bare ValueError is correct here. I traced the call chain — submit_concurrent_learning → workflow_concurrent_learning → make_naive_exploration_scheduler → make_calypso_task_group_from_config → set_params — and it all runs client-side during dpgen2 submit, before wf.submit(), not inside a dflow OP. FatalError would be wrong at this layer.